88 research outputs found
Learning to Detect Violent Videos using Convolutional Long Short-Term Memory
Developing a technique for the automatic analysis of surveillance videos in
order to identify the presence of violence is of broad interest. In this work,
we propose a deep neural network for the purpose of recognizing violent videos.
A convolutional neural network is used to extract frame level features from a
video. The frame level features are then aggregated using a variant of the long
short term memory that uses convolutional gates. The convolutional neural
network along with the convolutional long short term memory is capable of
capturing localized spatio-temporal features which enables the analysis of
local motion taking place in the video. We also propose to use adjacent frame
differences as the input to the model thereby forcing it to encode the changes
occurring in the video. The performance of the proposed feature extraction
pipeline is evaluated on three standard benchmark datasets in terms of
recognition accuracy. Comparison of the results obtained with the state of the
art techniques revealed the promising capability of the proposed method in
recognizing violent videos.Comment: Accepted in International Conference on Advanced Video and Signal
based Surveillance(AVSS 2017
LSTA: Long Short-Term Attention for Egocentric Action Recognition
Egocentric activity recognition is one of the most challenging tasks in video
analysis. It requires a fine-grained discrimination of small objects and their
manipulation. While some methods base on strong supervision and attention
mechanisms, they are either annotation consuming or do not take spatio-temporal
patterns into account. In this paper we propose LSTA as a mechanism to focus on
features from spatial relevant parts while attention is being tracked smoothly
across the video sequence. We demonstrate the effectiveness of LSTA on
egocentric activity recognition with an end-to-end trainable two-stream
architecture, achieving state of the art performance on four standard
benchmarks.Comment: Accepted to CVPR 201
Attention is All We Need: Nailing Down Object-centric Attention for Egocentric Activity Recognition
In this paper we propose an end-to-end trainable deep neural network model
for egocentric activity recognition. Our model is built on the observation that
egocentric activities are highly characterized by the objects and their
locations in the video. Based on this, we develop a spatial attention mechanism
that enables the network to attend to regions containing objects that are
correlated with the activity under consideration. We learn highly specialized
attention maps for each frame using class-specific activations from a CNN
pre-trained for generic image recognition, and use them for spatio-temporal
encoding of the video with a convolutional LSTM. Our model is trained in a
weakly supervised setting using raw video-level activity-class labels.
Nonetheless, on standard egocentric activity benchmarks our model surpasses by
up to +6% points recognition accuracy the currently best performing method that
leverages hand segmentation and object location strong supervision for
training. We visually analyze attention maps generated by the network,
revealing that the network successfully identifies the relevant objects present
in the video frames which may explain the strong recognition performance. We
also discuss an extensive ablation analysis regarding the design choices.Comment: Accepted to BMVC 201
A Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval
Every hour, huge amounts of visual contents are posted on social media and
user-generated content platforms. To find relevant videos by means of a natural
language query, text-video retrieval methods have received increased attention
over the past few years. Data augmentation techniques were introduced to
increase the performance on unseen test examples by creating new training
samples with the application of semantics-preserving techniques, such as color
space or geometric transformations on images. Yet, these techniques are usually
applied on raw data, leading to more resource-demanding solutions and also
requiring the shareability of the raw data, which may not always be true, e.g.
copyright issues with clips from movies or TV series. To address this
shortcoming, we propose a multimodal data augmentation technique which works in
the feature space and creates new videos and captions by mixing semantically
similar samples. We experiment our solution on a large scale public dataset,
EPIC-Kitchens-100, and achieve considerable improvements over a baseline
method, improved state-of-the-art performance, while at the same time
performing multiple ablation studies. We release code and pretrained models on
Github at https://github.com/aranciokov/FSMMDA_VideoRetrieval.Comment: Accepted for presentation at 30th ACM International Conference on
Multimedia (ACM MM
Data augmentation techniques for the Video Question Answering task
Video Question Answering (VideoQA) is a task that requires a model to analyze
and understand both the visual content given by the input video and the textual
part given by the question, and the interaction between them in order to
produce a meaningful answer. In our work we focus on the Egocentric VideoQA
task, which exploits first-person videos, because of the importance of such
task which can have impact on many different fields, such as those pertaining
the social assistance and the industrial training. Recently, an Egocentric
VideoQA dataset, called EgoVQA, has been released. Given its small size, models
tend to overfit quickly. To alleviate this problem, we propose several
augmentation techniques which give us a +5.5% improvement on the final accuracy
over the considered baseline.Comment: 16 pages, 5 figures; to be published in Egocentric Perception,
Interaction and Computing (EPIC) Workshop Proceedings, at ECCV 202
LSTA: Long Short-Term Attention for Egocentric Action Recognition
Egocentric activity recognition is one of the most challenging tasks in video analysis. It requires a fine-grained discrimination of small objects and their manipulation. While some methods base on strong supervision and attention mechanisms, they are either annotation consuming or do not take spatio-temporal patterns into account. In this paper we propose LSTA as a mechanism to focus on features from spatial relevant parts while attention is being tracked smoothly across the video sequence. We demonstrate the effectiveness of LSTA on egocentric activity recognition with an end-to-end trainable two-stream architecture, achieving state-of-the-art performance on four standard benchmarks
Convolutional Long Short-Term Memory Networks for Recognizing First Person Interactions
We present a novel deep learning approach for addressing the problem of interaction recognition from a first person perspective. The approach uses a pair of convolutional neural networks, whose parameters are shared, for extracting frame level features from successive frames of the video. The frame level features are then aggregated using a convolutional long short-term memory. The final hidden state of the convolutional long short-term memory is used for classification in to the respective categories. In our network the spatio-temporal structure of the input is preserved till the very final processing stage. Experimental results show that our method outperforms the state of the art on most recent first person interactions datasets that involve complex ego-motion. On UTKinect, it competes with methods that use depth image and skeletal joints information along with RGB images, while it surpasses previous methods that use only RGB images by more than 20% in recognition accuracy
- …